Papers with contextualized word embeddings
PiC: A Phrase-in-Context Dataset for Phrase Understanding and Semantic Search (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing benchmarks for phrase-similarity compare phrases alone (without context) and phrases with context (with or without context). |
| Approach: | They propose to use a dataset of 28K noun phrases accompanied by their contextual Wikipedia pages to train machine phrase embeddings. |
| Outcome: | The proposed dataset improves ranking-models’ accuracy and pushes span selection models near human accuracy, which is 95% Exact Match (EM) on semantic search given a query phrase and a passage. |
KeywordScape: Visual Document Exploration using Contextualized Keyword Embeddings (2022.emnlp-demos)
Copied to clipboard
| Challenge: | Existing tools for document visualization assume that keywords have static meanings, but contextualized word embeddings are unrealistic. |
| Approach: | They propose a visual exploration tool that visualizes contextualized word embeddings in documents based on keywords. |
| Outcome: | The proposed tool visualizes keywords in terms of their contextualized embeddings in a semantic landscape that keeps keywords with similar context close to each other, allowing for a more precise search and comparison of documents. |
Leveraging Three Types of Embeddings from Masked Language Models in Idiom Token Classification (2022.starsem-1)
Copied to clipboard
| Challenge: | Recent research shows that contextualized word embeddings can give promising results for idiom token classification. |
| Approach: | They propose to leverage contextualized word embeddings from masked language models to improve idiom token classification. |
| Outcome: | The proposed method improves idiom token classification for English and Japanese datasets. |
Contextualized context2vec (D19-55)
Copied to clipboard
| Challenge: | Lexical substitution ranks substitution candidates from the viewpoint of paraphrasability for a target word in a given sentence. |
| Approach: | They propose a method that combines two approaches to contextualize word embeddings for lexical substitution. |
| Outcome: | The proposed method outperforms the current state-of-the-art method and assigns English proficiency levels to all target words and substitution candidates. |
Word class flexibility: A deep contextualized approach (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on word class flexibility have been fraught with difficulties in quantifying it accurately and at scale. |
| Approach: | They propose a method to quantify word class flexibility in 37 languages using contextualized word embeddings. |
| Outcome: | The proposed method builds on recent work in contextualized word embeddings to quantify semantic shift between word classes and uncovers shared tendencies in class flexibility across languages. |
Semantic Frame Induction using Masked Word Embeddings and Two-Step Clustering (2021.acl-short)
Copied to clipboard
| Challenge: | Recent studies show that clustering-based methods focus too much on the surface information of frame-evoking verbs and divide instances of the same verb into too many different frame clusters. |
| Approach: | They propose a semantic frame induction method using masked word embeddings and two-step clustering to overcome these drawbacks. |
| Outcome: | The proposed method reduces the number of instances of the same verb into too many clusters . it uses masked word embeddings and two-step clustering to avoid drawbacks compared with other methods . |
Semantic Frame Induction with Deep Metric Learning (2023.eacl-main)
Copied to clipboard
| Challenge: | Recent studies have shown the usefulness of contextualized word embeddings in semantic frame induction, but they are not always consistent with human intuitions about semantic frames. |
| Approach: | They propose a model that fine-tunes contextualized embeddings to perform semantic frame induction. |
| Outcome: | The proposed model improves clustering evaluation scores on FrameNet by 8 points or more. |
Combining Self-Training and Self-Supervised Learning for Unsupervised Disfluency Detection (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to disfluency detection rely on human annotations, which are expensive to obtain. |
| Approach: | They propose an unsupervised learning paradigm which can work with unlabeled text corpora. |
| Outcome: | The proposed method performs better than existing supervised systems using word embeddings. |
Paths to Relation Extraction through Semantic Structure (2021.findings-acl)
Copied to clipboard
| Challenge: | Syntactic and semantic structure directly reflect relations expressed by the text at hand and are therefore very useful for relation extraction (RE) |
| Approach: | They propose two methods for integrating broad-coverage semantic structure into supervised RE models by encoding semantic DAGs. |
| Outcome: | The proposed methods overshadow the use of syntactic integrations in RE . they reduce UCCA into a bilexical structure and encode semantic DAG structures . |
Entity-Centric Contextual Affective Analysis (P19-1)
Copied to clipboard
| Challenge: | Existing methods for analyzing people portrayals take an unsupervised approach, or rely on domain-specific knowledge. |
| Approach: | They show how contextualized word embeddings can be used to capture affect dimensions in portrayals of people. |
| Outcome: | The proposed method can capture affect dimensions in portrayals of men and women . it is biased towards training data, which limits its usefulness to in-domain analyses . |
Detecting Unassimilated Borrowings in Spanish: An Annotated Corpus and Approaches to Modeling (2022.acl-long)
Copied to clipboard
| Challenge: | a corpus of Spanish newswire rich in unassimilated lexical borrowings is used to identify the language of a word. |
| Approach: | They propose to annotate a corpus of Spanish newswire rich in unassimilated lexical borrowings and evaluate how models perform on this task. |
| Outcome: | The proposed model outperforms models fed with subword embeddings and Transformer-based embeddables on the Spanish newswire corpus. |
FOOL ME IF YOU CAN! An Adversarial Dataset to Investigate the Robustness of LMs in Word Sense Disambiguation (2024.emnlp-main)
Copied to clipboard
Mohamad Ballout, Anne Dedert, Nohayr Abdelmoneim, Ulf Krumnack, Gunther Heidemann, Kai-Uwe Kühnberger
| Challenge: | Word sense disambiguation (WSD) is a key task in natural language processing . however, these models struggle with recognizing semantic boundaries in adversarial contexts . |
| Approach: | They propose to use a coarse-grained WSD dataset to assess model robustness . they found that some models struggled to correctly disambiguate homonyms in adversarial contexts . |
| Outcome: | The proposed dataset includes four test sets to assess the robustness of language models in WSD tasks. |
Improving Disfluency Detection by Self-Training a Self-Attentive Model (2020.acl-main)
Copied to clipboard
| Challenge: | Existing self-attentive parsers using contextualized word embeddings produce state-of-the-art results in joint parsing and disfluency detection. |
| Approach: | They propose to use contextualized word embeddings to train a neural model using unlabeled data to train parsers. |
| Outcome: | The proposed method produces state-of-the-art results in parsing and disfluency detection in speech transcripts. |
One Word, Two Sides: Traces of Stance in Contextualized Word Representations (2022.coling-1)
Copied to clipboard
| Challenge: | a Lexical Semantic Change study examines the way we use words . it focuses on the use of words by people who disagree on a particular topic . |
| Approach: | They examine whether word embeddings reflect the way we use words . they use BERT embeddables from datasets with stance annotations to examine this question . |
| Outcome: | The results show that people with opposing stances use different words when talking about a topic . the results are not related to studies that investigate the usage of specific words across different viewpoints. |
CWTM: Leveraging Contextualized Word Embeddings from BERT for Neural Topic Modeling (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing topic models rely on bag-of-words (BOW) representations to capture word order information. |
| Approach: | They propose a neural topic model that integrates contextualized word embeddings from BERT to learn the topic vector of a document without BOW information. |
| Outcome: | The proposed model generates more coherent and meaningful topics compared to existing models while accommodating unseen words in newly encountered documents. |
Unsupervised Cross-Lingual Part-of-Speech Tagging for Truly Low-Resource Scenarios (2020.emnlp-main)
Copied to clipboard
| Challenge: | a limited set of translations into one or more high-resource languages are available for POS tagging . a bi-LSTM architecture that uses contextualized word embeddings improves performance . |
| Approach: | They propose an unsupervised cross-lingual transfer approach for part-of-speech tagging . they use the Bible as parallel data to learn POS taggers for target languages . |
| Outcome: | The proposed approach improves accuracy on 12 diverse languages . the Bible is used as a parallel corpus for the study . |
Evaluating Contextualized Representations of (Spanish) Ambiguous Words: A New Lexical Resource and Empirical Analysis (2025.naacl-long)
Copied to clipboard
| Challenge: | Few studies have systematically compared LMs’ contextualized word embeddings for languages beyond English. |
| Approach: | They evaluate Spanish ambiguous nouns in context in a suite of Spanish-language monolingual and multilingual BERT-based models. |
| Outcome: | The proposed model captures some variance in human relatedness judgments but falls short of the human benchmark. |
VCDM: Leveraging Variational Bi-encoding and Deep Contextualized Word Representations for Improved Definition Modeling (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches for definition modeling combine distributional and lexical semantics in an implicit rather than direct way. |
| Approach: | They propose a model that introduces a continuous latent variable to model the relationship between a phrase and its definition. |
| Outcome: | The proposed model achieves state-of-the-art performance on four challenging benchmarks and the first non-English corpus. |
RAW-C: Relatedness of Ambiguous Words in Context (A New Lexical Resource for English) (2021.acl-long)
Copied to clipboard
| Challenge: | lexical ambiguity is a problem for NLP, but few tasks evaluate its impact on human intuitions. |
| Approach: | They propose to use contextualized word embeddings to evaluate word meaning . they use a dataset of human relatedness judgments and human estimates of sense dominance . |
| Outcome: | The proposed model matches human intuitions with contextualized embeddings on 112 ambiguous words in context with 672 sentence pairs. |
An Improved Neural Baseline for Temporal Relation Extraction (D19-1)
Copied to clipboard
| Challenge: | Existing datasets are small and/or have low inter-annotator agreements. |
| Approach: | They propose a new neural system that achieves 10% absolute accuracy improvement over the previous best system. |
| Outcome: | The proposed system achieves 10% absolute improvement over the previous best system on two benchmark datasets. |
Contextualized Embeddings based Transformer Encoder for Sentence Similarity Modeling in Answer Selection Task (2020.lrec-1)
Copied to clipboard
| Challenge: | Word embeddings that consider context have attracted great attention for natural language processing tasks in recent years. |
| Approach: | They propose two different approaches to integrate contextualized word embeddings with transformer encoders for sentence similarity modeling. |
| Outcome: | The proposed model outperforms the feature-based approach on six datasets. |